[develop] Make `get_obs` tasks day-dependent in workflow; other improvements and bug fixes #1137

gsketefian · 2024-10-08T16:59:38Z

DESCRIPTION OF CHANGES:

This PR fixes multiple bugs in the verification (vx) and other parts of the SRW App, the main one being that the get_obs tasks as well as some of the vx pre-processing tasks currently do not work for an experiment with multiple cycles if those cycles overlap in time (bug discovered by @michelleharrold and @willmayfield). Fixes and changes made by this PR are described in more detail below.

Changes related to `get_obs` tasks:

Make get_obs tasks in the ROCOTO workflow obs-day-based as opposed to cycle-based. Thus, for each day for which obs are needed for vx (and for each obs type that is needed for vx), there is now a get_obs workflow task.
Move the functionality in the ex-script exregional_get_verif_obs.sh to the new python script get_obs.py. The new exregional_get_verif_obs.sh is now a very short script that just calls get_obs.py.
The new get_obs.py script, along with changes to setup.py to calculate the times at which various types of obs need to be retrieved, ensure that no clobbering of retrieved obs files occurs (this currently does occur if cycles overlap).
In config_defaults.yaml, introduce new variables specifying the obs availability interval for each of the four obs types (CCPA, NOHRSC, MRMS, NDAS) that might be retrieved. These variables are [CCPA|NOHRSC|MRMS|NDAS]_OBS_AVAIL_INTVL_HRS.
setup.py now checks that multiple consistency constraints and requirements on the temporal vx parameters in the SRW configuration file (e.g. the accumulation periods, the obs availability intervals, the forecast output interval) are satisfied that would otherwise cause errors in the workflow. (setup.py calls functions in the (renamed) script set_cycle_and_obs_timeinfo.py to run these checks.) If such inconsistencies exist, the parameters are either adjusted to fix them or, if that is not possible, the experiment generation process is stopped.
In config_defaults.yaml, introduce flags that determine whether or not to delete the raw obs directories and files that the get_obs tasks create after the raw obs have been copied/moved/renamed to their final/processed locations. These new flags are REMOVE_RAW_OBS_[CCPA|NOHRSC|MRMS|NDAS].
In config_defaults.yaml, move the base directories for the obs, i.e. [CCPA|NOHRSC|MRMS|NDAS]_OBS_DIR, from the platform section to the verification section so that they are near the METplus obs file name templates (OBS_...FN_TEMPLATE) for which they serve as base directories.
The processed/final files that the get_obs tasks create are now located and named as specified by the combination of the obs base directory (e.g. CCPA_OBS_DIR) and the obs file name template (e.g. CCPA_APCP_FN_TEMPLATE). Currently, the processed/final file that the get_obs tasks first look for are, say for CCPA, {CCPA_OBS_DIR}/{CCPA_APCP_FN_TEMPLATE}, but if these files don't exist and the obs need to be retrieved, the retrieved and processed file names are not necessarily given by this template. With this PR, the raw files are renamed and moved after retrieval to ensure they are located at {..._OBS_DIR}/{..._FN_TEMPLATE}.
Retrieve only 6-hourly NOHRSC snow accumulation obs, not 24-hourly accumulations. Currently, 24-hour accumulated obs are also retrieved (although there doesn't seem to be a WE2E test for it).
Modify the configuration file parm/data_locations.yml for retrieve_data.py to extract all files in an archive at a time (i.e. per call to retrieve_data.py) instead of extracting only one obs file out of an archive for each call to retrieve_data.py. This speeds up the data retrieval significantly since a large portion of the get_obs tasks' wallclock time is spent establishing a connection to HPSS.
Modify parm/data_locations.yml to account for the change in prebpufr (NDAS) obs file names on May 22, 2024. This is currently causing get_obs_ndas tasks to fail for cycles at or after this date. (Bug found by @michelleharrold.)
Fix vx task dependencies to work with new obs-day-based get_obs tasks. Now, all get_obs tasks (i.e. for all obs days) for a given obs type must be complete before any vx tasks for that obs type can launch. This doesn't cause any significant delay because the get_obs tasks run in parallel and get at most one day's worth of obs.

Changes related to vx pre-processing tasks (`PcpCombine_obs` and `Pb2nc_obs`):

Add PcpCombine_obs tasks for both 6-hour and 24-hour accumulations of NOHRSC obs. The one for 6-hour accumulation simply converts the grib2 obs files to NetCDF, while the one for 24-hour accumulation adds the 6-hour grib2 obs to obtain a NetCDF file for 24-hour obs accumulations.
Place all output from PcpCombine_obs tasks (both for CCPA and NOHRSC) under the cycle directories, just as is done for the analogous PcpCombine_fcst tasks for forecasts. This is because accumulations, even for obs, depend on the start time of the cycle, e.g. 6-hour CCPA accumulations needed to verify a set of forecasts that start at 00Z will be different than 6-hour CCPA accumulations needed to verify a set of forecasts that start at 03Z. (Currently, the output files from these tasks are placed in the metprd directory under the main experiment directory without consideration for the start times of the accumulations.)
Make the Pb2nc_obs task for NDAS obs-day-dependent (unlike the PcpCombine_obs tasks, which are cycle-dependent). This can be done because unlike accumulations, the result of the Pb2nc_obs task does not depend on the starting time of the cycles; it only depends on a given valid time. Also, keep the output of the Pb2nc_obs task in the cycle-independent directory metprd directly under the main experiment directory.

Small, self-contained bug fixes and improvements:

Move evaluation of METplus time strings out of what used to be set_vx_fhr_list.sh (now renamed to set_leadhrs.sh) and into a new bash script (bash_utils/eval_METplus_timestr_tmpl.sh) to make it easier to change this functionality to python later on.
Allow WE2E test names to include dots since dots (like underscores) are handy to use as separators in the test name.
Add the two new SRW config parameters VX_CONFIG_[DET|ENS]_FN in config_defaults.yaml that specify the yaml configuration files to use for deterministic and ensemble verification. The default values for these are the files vx_config_[det|ens]_fn.yaml in parm/metplus. These parameters allow a user to specify other user-created yaml files in this directory to use for the vx configuration so that the default files, which are under version control, do not have to be changed.
Change some metatask and task names for clarity and consistency.
Add an option to mrms_pull_topofhour.py to not assume that there is a valid-date subdirectory under the specified source directory and to not add such a subdirectory under the specified output directory when generating output. This is handy when calling this script from the new get_obs.py script.
Fix bug in parm/wflow/verify_det.yaml so that all tasks have a cycldefs statement by default. This bug was causing GridStat workflow tasks for CCPA and NOHRSC obs to be created for cycles not defined for the workflow (these extraneous cycles probably correspond to the default set of cycles that a task gets assigned by ROCOTO when it does not contain an explicit cycledefs statement). (Bug found by @michelleharrold, solution by @mkavulich.)
Fix bug in scripts/exregional_run_met_gridstat_or_pointstat.sh to append a string for the cycle date ("_YYYYMDDHH") to the name of the metplus log file for deterministic GridStat and PointStat tasks. This was causing the metplus log file for GridStat for a given cycle tasks to be overwritten by those for other cycles. (Bug found by @michelleharrold.)
Fix bug in parm/default_workflow.yaml in "cycled_from_second" section in which the starting YYYYMMDDHH value of the cycledef can contain an HH value that is larger than 23. This currently happens because this HH is obtained directly from INCR_CYCL_FREQ without checking whether that value is less than 24.
Fix bug In launch_FV3LAM_wflow.sh to change double quotes to single quotes to prevent failure in the interpretation of the command by cron.

New WE2E tests added:

The following new WE2E tests were added to the verification subdirectory under test_configs to test various aspects of the new code:

get_obs_hpss.do_vx_det.multicyc.cycintvl_07hr_inits_vary_fcstlen_09hr.ncep-hrrr
get_obs_hpss.do_vx_det.multicyc.cycintvl_11hr_inits_vary_fcstlen_03hr.ncep-hrrr
get_obs_hpss.do_vx_det.multicyc.cycintvl_24hr_inits_00z_fcstlen_03hr.ncep-hrrr
get_obs_hpss.do_vx_det.multicyc.cycintvl_24hr_inits_12z_fcstlen_03hr.nssl-mpas
get_obs_hpss.do_vx_det.multicyc.cycintvl_24hr_inits_12z_fcstlen_48hr.nssl-mpas
get_obs_hpss.do_vx_det.multicyc.cycintvl_24hr_inits_21z_fcstlen_03hr.ncep-hrrr
get_obs_hpss.do_vx_det.multicyc.cycintvl_96hr_inits_12z_fcstlen_48hr.nssl-mpas
get_obs_hpss.do_vx_det.singlecyc.init_00z_fcstlen_36hr.winter_wx.SRW

The purpose of each of these is described in the description section of the corresponding test config file.

Type of change

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update

TESTS CONDUCTED:

Three sets of WE2E tests were conducted on Hera/intel:

The "fundamental" suite consisting of:
- grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2
- grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v17_p8_plot
- grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_GFS_v16
- grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_HRRR_suite_HRRR
- grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta
- grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_RAP_suite_WoFS_v0
The existing verification tests, consisting of:
- `MET_ensemble_verification
- MET_ensemble_verification_only_vx
- MET_ensemble_verification_only_vx_time_lag
- MET_ensemble_verification_winter_wx
- MET_verification
- MET_verification_only_vx
- MET_verification_winter_wx
The newly added get_obs/verification tests, consisting of:
- get_obs_hpss.do_vx_det.multicyc.cycintvl_07hr_inits_vary_fcstlen_09hr.ncep-hrrr
- get_obs_hpss.do_vx_det.multicyc.cycintvl_11hr_inits_vary_fcstlen_03hr.ncep-hrrr
- get_obs_hpss.do_vx_det.multicyc.cycintvl_24hr_inits_00z_fcstlen_03hr.ncep-hrrr
- get_obs_hpss.do_vx_det.multicyc.cycintvl_24hr_inits_12z_fcstlen_03hr.nssl-mpas
- get_obs_hpss.do_vx_det.multicyc.cycintvl_24hr_inits_12z_fcstlen_48hr.nssl-mpas
- get_obs_hpss.do_vx_det.multicyc.cycintvl_24hr_inits_21z_fcstlen_03hr.ncep-hrrr
- get_obs_hpss.do_vx_det.multicyc.cycintvl_96hr_inits_12z_fcstlen_48hr.nssl-mpas
- get_obs_hpss.do_vx_det.singlecyc.init_00z_fcstlen_36hr.winter_wx.SRW

All tests were successful.

DOCUMENTATION:

I am not familiar with the new RST file setup and will need help moving my documentation from comments in the code to the RST files.

CHECKLIST

My code follows the style guidelines in the Contributor's Guide
I have performed a self-review of my own code using the Code Reviewer's Guide
I have commented my code, particularly in hard-to-understand areas
My changes need updates to the documentation. I have made corresponding changes to the documentation
Possibly; I'm not sure what exactly the documentation requirements are currently.
My changes do not require updates to the documentation (explain).
My changes generate no new warnings
New and existing tests pass with my changes
Any dependent changes have been merged and published

LABELS (optional):

A Code Manager needs to add the following labels to this PR:

CONTRIBUTORS (optional):

@michelleharrold @mkavulich @JeffBeck-NOAA @willmayfield

…the tar file where the prepbufr files live changed"

…y Michelle Harrold, solution by Michael Kavulich.

…ntStat tasks' METplus log files.

…ing cycles for CCPA and MRMS but not yet for NDAS or NOHRSC.

…thout performing unnecessary repeated pulls.

… comments).

… they're per-cycle or per-day.

…nup and comments.

…files from HPSS (and works with multiple cycles).

…e cleanup is happening.

…les, that are expected to be created once the task is finished actually get created. This is needed because it is possible that for some forecast hours for which there is overlap between cycles, the files are being retrieved and processed by the get_obs_... task for another cycle.

…nd EnsembleStat tasks such that GenEnsProd does not depend on the completion of get_obs_... tasks (because it doesn't need observations) but only forecast output while EnsembleStat does.

…d due to changes to dependencies of GenEnsProd tasks in previous commit(s).

…tending to time out for 48-hour forecasts.

…sure PcpCombine operates only on those hours unique to the cycle, i.e. for those times starting from the initial time of the cycle to just before the initial time of the next cycle. For the PcpCombine_obs task for the last cycle, allow it to operate on all hours of that cycle's forecast. This ensures that the PcpCombine tasks for the various cycles do not clobber each other's output. Accordingly, change the dependencies of downstream tasks that depend on PcpCombine obs output to make sure they include all PcpCombine_obs tasks that cover the forecast period of the that downstream task's cycle.

…ossibly also get_obs_ndas by putting in sleep commands.

Co-authored-by: Gillian Petro <[email protected]>

doc/UsersGuide/BuildingRunningTesting/RunSRW.rst

Co-authored-by: Gillian Petro <[email protected]>

…asks' into feature/daily_obs_tasks

gspetro-NOAA · 2024-11-13T21:47:22Z

@gsketefian I noticed that the tech doc test is failing. You can follow the instructions in the slides here or in the documentation to update. I'm hoping it will be pretty straightforward now that I've put together the instructions, but let me know if you need help.

MichaelLueken · 2024-11-13T21:56:09Z

The verification WE2E tests have successfully passed on Hera Intel:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used
----------------------------------------------------------------------------------------------------
vx-det_multicyc_fcst-overlap_ncep-hrrr_20241113161443              COMPLETE               3.69
vx-det_multicyc_long-fcst-overlap_nssl-mpas_20241113161445         COMPLETE              12.29
MET_verification_only_vx_20241113161447                            COMPLETE               0.28
vx-det_multicyc_first-obs-00z_ncep-hrrr_20241113161449             COMPLETE               0.78
vx-det_multicyc_last-obs-00z_ncep-hrrr_20241113161450              COMPLETE               0.85
vx-det_multicyc_no-fcst-overlap_ncep-hrrr_20241113161452           COMPLETE               1.89
MET_verification_20241113161454                                    COMPLETE              14.84
vx-det_multicyc_long-fcst-no-overlap_nssl-mpas_20241113161455      COMPLETE              13.13
MET_ensemble_verification_only_vx_time_lag_20241113161457          COMPLETE               4.33
MET_ensemble_verification_20241113161500                           COMPLETE              47.35
vx-det_multicyc_no-00z-obs_nssl-mpas_20241113161502                COMPLETE               0.78
MET_verification_winter_wx_20241113161504                          COMPLETE              24.39
vx-det_long-fcst_custom-vx-config_gfs_20241113161506               COMPLETE               0.70
vx-det_long-fcst_custom-vx-config_aiml-fourcastnet_20241113161507  COMPLETE               0.64
vx-det_long-fcst_custom-vx-config_aiml-graphcast_20241113161509    COMPLETE               0.59
vx-det_long-fcst_custom-vx-config_aiml-panguweather_2024111316151  COMPLETE               0.63
MET_ensemble_verification_winter_wx_20241113161513                 COMPLETE             160.75
vx-det_long-fcst_winter-wx_SRW-staged_20241113161515               COMPLETE               1.06
MET_ensemble_verification_only_vx_20241113161517                   COMPLETE               1.15
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE             290.12

as well as the verification tests on Hera GNU:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used
----------------------------------------------------------------------------------------------------
vx-det_multicyc_fcst-overlap_ncep-hrrr_20241113173041              COMPLETE               7.32
vx-det_multicyc_long-fcst-overlap_nssl-mpas_20241113173044         COMPLETE              26.93
MET_verification_only_vx_20241113173045                            COMPLETE               0.31
vx-det_multicyc_first-obs-00z_ncep-hrrr_20241113173047             COMPLETE               1.30
vx-det_multicyc_last-obs-00z_ncep-hrrr_20241113173048              COMPLETE               1.39
vx-det_multicyc_no-fcst-overlap_ncep-hrrr_20241113173050           COMPLETE               3.17
MET_verification_20241113173052                                    COMPLETE              18.97
vx-det_multicyc_long-fcst-no-overlap_nssl-mpas_20241113173053      COMPLETE              27.87
MET_ensemble_verification_only_vx_time_lag_20241113173055          COMPLETE              12.94
MET_ensemble_verification_20241113173058                           COMPLETE              72.61
vx-det_multicyc_no-00z-obs_nssl-mpas_20241113173059                COMPLETE               1.30
MET_verification_winter_wx_20241113173101                          COMPLETE              32.84
vx-det_long-fcst_custom-vx-config_gfs_20241113173102               COMPLETE               0.79
vx-det_long-fcst_custom-vx-config_aiml-fourcastnet_20241113173104  COMPLETE               0.78
vx-det_long-fcst_custom-vx-config_aiml-graphcast_20241113173106    COMPLETE               0.78
vx-det_long-fcst_custom-vx-config_aiml-panguweather_2024111317310  COMPLETE               0.77
MET_ensemble_verification_winter_wx_20241113173110                 COMPLETE             212.75
vx-det_long-fcst_winter-wx_SRW-staged_20241113173112               COMPLETE               1.41
MET_ensemble_verification_only_vx_20241113173114                   COMPLETE               1.55
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE             425.78

…for each new module was added in a previous commit).

… the glossary.

… of new python functions; add type for each argument and return value; use latex-style math formatting for equations; other minor formatting adjustments.

gsketefian · 2024-11-14T11:51:42Z

@gspetro-NOAA I fixed up as much of the docs as I could, but I ran into one issue that I don't have a good fix for. The problem is that the scripts get_obs.py and eval_metplus_timestr_tmpl.py (and, indirectly, set_leadhrs.py) need the environment variable METPLUS_ROOT to import a module from METplus, but METPLUS_ROOT is not available when making the docs (maybe we have to load the modules for running vx). First, I got errors like:

WARNING: autodoc: failed to import module 'eval_metplus_timestr_tmpl'; the following exception was raised:
Traceback (most recent call last):
  File "/home/Gerard.Ketefian/miniconda3/lib/python3.12/site-packages/sphinx/ext/autodoc/importer.py", line 58, in import_module
    return importlib.import_module(modname)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/Gerard.Ketefian/miniconda3/lib/python3.12/importlib/__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 995, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/scratch2/BMC/fv3lam/Gerard.Ketefian/AGILE3/ufs-srweather-app/ush/eval_metplus_timestr_tmpl.py", line 7, in <module>
    sys.path.append(os.environ['METPLUS_ROOT'])
                    ~~~~~~~~~~^^^^^^^^^^^^^^^^
  File "<frozen os>", line 685, in __getitem__
KeyError: 'METPLUS_ROOT'

Then I thought to define METPLUS_ROOT in Makefile itself (for the specific case of Hera; see my most recent changes in Makefile). That got me around finding METplus, but now I get errors like this:

WARNING: autodoc: failed to import module 'eval_metplus_timestr_tmpl'; the following exception was raised:
No module named 'dateutil'

I decided to stop there and see if you have any ideas. I'll tag @mkavulich too since he's the original author of the use of METPLUS_ROOT in eval_metplus_timestr_tmpl.py.

gsketefian · 2024-11-14T15:26:07Z

@MichaelLueken FYI that I made that change of the wallclock time in verify_ens.yaml. I'm now waiting to hear back from @gspetro-NOAA and/or @mkavulich regarding a fix to the doc build issue I ran into (see my previous message). Thanks.

MichaelLueken · 2024-11-14T18:19:27Z

@gsketefian and @gspetro-NOAA -

Unfortunately, it looks like the modification to doc/Makefile won't work for RtD and the GHA test. In the Doc Tests GHA, I see the following in the Build documentation section:

WARNING: autodoc: failed to import module 'eval_metplus_timestr_tmpl'; the following exception was raised:
No module named 'metplus' [autodoc.import_object]
WARNING: autodoc: failed to import module 'get_obs'; the following exception was raised:
No module named 'metplus' [autodoc.import_object]
WARNING: autodoc: failed to import module 'set_leadhrs'; the following exception was raised:
No module named 'metplus' [autodoc.import_object]

Additionally, the technical documentation for the above three scripts in Read the Docs are empty.

The modification made to doc/Makefile will allow the documentation to find metplus on Hera, but not on any other machine or on the GitHub runner, leading to the warning message.

In the new python scripts, I see:

try:
    sys.path.append(os.environ['METPLUS_ROOT'])
except:
    print("\nERROR ERROR ERROR\n")
    print("Environment variable METPLUS_ROOT must be set to use this script\n")
    raise
from metplus.util import string_template_substitution as sts

Since there is now a dependency on METplus, will we need to include both MET and METplus in the environment used to run the Doc Tests GHA?

It looks like dateutil is a METplus requirement. So, if MET and METplus are required, then python-dateutil will also be required within the build environment for technical documentation.

gspetro-NOAA · 2024-11-14T18:51:00Z

@MichaelLueken @gsketefian Is there a way to put the try/except statement under if __name__ == "__main__":? @maddenp helped me with a similar error. Based on what he said, Sphinx's autodoc imports the Python module it is trying to produce docs for. When import happens, all the top-level code (outside of functions, classes, etc.) has to be executed, so lines like

try:
    sys.path.append(os.environ['METPLUS_ROOT'])
except:
    print("\nERROR ERROR ERROR\n")
    print("Environment variable METPLUS_ROOT must be set to use this script\n")
    raise
from metplus.util import string_template_substitution as sts

get executed. This causes an error precisely because the 'METPLUS_ROOT' directory is not available. Putting this code under if __name__ == "__main__": might be one way to solve the problem. It seems like you can put the try/except block there and leave the import statement below where it is. (Note that you'll likely have to do this for every script where the try/except statement is outside if __name__ == "__main__":.) If the import statement alone causes a problem, it can be added to the autodoc_mock_imports list in conf.py.

maddenp-noaa · 2024-11-14T19:33:51Z

Your linter might complain about an import statement somewhere other than at the top of the file, but there are good reasons to suppress linter complaints and this might be one of them.

Alternatively, with from importlib import import_module at the top of the module, you can do sts = import_module("metplus.util.string_template_substitution") close to where sts is needed to do a just-in-time dynamic import.

MichaelLueken · 2024-11-14T21:46:35Z

Thanks, @gspetro-NOAA and @maddenp-noaa, for options to try to get the technical documentation to work!

I think the main issue here is that it is impossible to set metplus if the environmental variable, METPLUSROOT, isn't available. The modification that @gsketefian made to doc/Makefile allows us to get around the environmental variable issue, but then metplus isn't available.

As @gspetro-NOAA noted, adding metplus to the autodoc_mock_imports list in conf.py allowed the build to successfully complete.

@gsketefian -

In doc/conf.py, please add metplus to the autodoc_mock_imports list (line 255):

autodoc_mock_imports = ["f90nml","cartopy","mpl_toolkits.basemap","fill_jinja_template",
   "matplotlib","numpy","uwtools","mpl_toolkits","metplus",
   ]

This is how I was able to get around the issue with metplus and dateutil.

doc/Makefile

…ile for sphinx (conf.py) (this will give sphinx access to METplus); remove definition of METPLUS_ROOT as an environment variable from the Makefile for the docs and instead define it in conf.py (as just a null string since it isn't actually used to load METplus).

gsketefian · 2024-11-15T16:29:59Z

@MichaelLueken @gspetro I followed your suggestion of adding "metplus" to autodoc_mock_imports in conf.py, and that worked. Also, I restored theMakefile for the documentation to its original version and instead defined the environment variable METPLUS_ROOT in conf.py. There, I set it to just a blank string since it just needs to exist; it's not used directly to load METplus (its contents get added to the system path, but that path already contains the location of METplus due to the addition of "metplus" to autodoc_mock_imports). Hopefully the doc test will succeed now.

MichaelLueken

@gsketefian -

Thank you for addressing the documentation issues! The Test Docs GHA test is passing now and the technical documentation for the new ush/ python scripts are being generated in RtD!

Since the new verification WE2E tests are successfully running on Hera GNU and Hera Intel, I will approve these changes and start the automated testing.

Co-authored-by: Gillian Petro <[email protected]>

MichaelLueken · 2024-11-18T15:50:16Z

The rerun of the Jenkins tests on Derecho have successfully completed:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used 
----------------------------------------------------------------------------------------------------
custom_ESGgrid_IndianOcean_6km_20241118074436                      COMPLETE              41.66
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_plot_20  COMPLETE              86.90
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2_20241  COMPLETE              15.68
grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_GFS_v16_2024111807444  COMPLETE              67.76
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta_2024  COMPLETE             744.00
grid_RRFS_CONUScompact_13km_ics_HRRR_lbcs_RAP_suite_HRRR_20241118  COMPLETE              66.49
grid_RRFS_CONUScompact_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_  COMPLETE              24.85
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta_2  COMPLETE              19.42
grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_HRRR_suite_HRRR_2024111807444  COMPLETE              38.33
pregen_grid_orog_sfc_climo_20241118074449                          COMPLETE              20.17
specify_template_filenames_20241118074451                          COMPLETE              20.97
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE            1146.23

The rest of the tests successfully passed on Friday evening.

Since the last conversations have been resolved, I will now move forward with merging this PR.

gsketefian · 2024-11-20T17:49:08Z

@MichaelLueken Thanks for shepherding this through!

gsketefian added 30 commits July 9, 2024 13:57

Bug fix to support the %H format in METplus via printf.

e97a46c

Bug fix to the bug fix!

815c941

Bug fix from Michelle H. for prepbufr files: "On May 22, the name of …

bc85480

…the tar file where the prepbufr files live changed"

Bug fix for removing phantom 00-hour tasks from workflow. Bug found b…

81d61b8

…y Michelle Harrold, solution by Michael Kavulich.

Bug fix: Append cycle date to names of deterministic GridStat and Poi…

35530ab

…ntStat tasks' METplus log files.

Version of ex-script for pulling obs that works for multiple overlapp…

6c548ce

…ing cycles for CCPA and MRMS but not yet for NDAS or NOHRSC.

Changes to make get_obs_mrms tasks to work for mulitple cycles and wi…

307f92e

…thout performing unnecessary repeated pulls.

Minor improvement for consistency.

be54216

New version of CCPA obs fetching (rename variables, include lots more…

af2ab4c

… comments).

Minor changes to ccpa section.

85c3d58

Changes for MRMS.

b7c6f00

Clean up comments in the MRMS section.

2bc8ed1

Minor fixes to NDAS section.

1845342

Change names of raw directories for CCPA and MRMS to indicate whether…

8c38c19

… they're per-cycle or per-day.

Version with NDAS changes that seems to work. Still need lots of clea…

7f53187

…nup and comments.

Second set of NDAS changes so that there are no repeat pulls of NDAS …

7926705

…files from HPSS (and works with multiple cycles).

Clean up NDAS section in get_obs_... ex-script.

f8c3ec6

Merge branch 'develop' into bugfix/vx_bundle

df07f82

Add debugging statement to clarify the current working directory wher…

bc276fe

…e cleanup is happening.

Fix the workflow task dependencies and ex-script for the GenEnsProd a…

dc4971d

…nd EnsembleStat tasks such that GenEnsProd does not depend on the completion of get_obs_... tasks (because it doesn't need observations) but only forecast output while EnsembleStat does.

Bug fixes after running WE2E vx suite.

13aba39

Bugfix to dependencies of ensemble vx tasks that come after GenEnsPro…

860f62e

…d due to changes to dependencies of GenEnsProd tasks in previous commit(s).

Bug fixes to get all WE2E vx tests to succeed.

e54ec16

Increase default wallclock time for get_obs_ccpa tasks since they're …

8e8a1c1

…tending to time out for 48-hour forecasts.

Bug fix in yaml.

5550a41

Fix still-existing problem of file clobbering with get_obs_mrms and p…

c76ed1a

…ossibly also get_obs_ndas by putting in sleep commands.

Improvements to jinja2 code to put in dependencies from other cycles.

3f1dea1

Bug fix.

53dd688

Update doc/UsersGuide/BuildingRunningTesting/RunSRW.rst

9bff39b

Co-authored-by: Gillian Petro <[email protected]>

gspetro-NOAA reviewed Nov 13, 2024

View reviewed changes

doc/UsersGuide/BuildingRunningTesting/RunSRW.rst Outdated Show resolved Hide resolved

gsketefian and others added 3 commits November 13, 2024 12:46

Update doc/UsersGuide/BuildingRunningTesting/RunSRW.rst

d2f31cb

Co-authored-by: Gillian Petro <[email protected]>

Merge branch 'develop' into feature/daily_obs_tasks

d878ecd

Merge remote-tracking branch 'refs/remotes/origin/feature/daily_obs_t…

9c6f914

…asks' into feature/daily_obs_tasks

gsketefian added 6 commits November 13, 2024 17:09

Add new terms to glossary.

f715f44

Remove rst file in TechDocs for a non-existent python script.

6b1cef9

Update list of modules to be included in the tech docs (the rst file …

8a3545b

…for each new module was added in a previous commit).

Bug fix: Change :ref: to :term: to reference the definition of DTC in…

c092798

… the glossary.

Fix indentation for argument and return value lists in the docstrings…

3d0bd5c

… of new python functions; add type for each argument and return value; use latex-style math formatting for equations; other minor formatting adjustments.

Define METPLUS_ROOT in the Makefile for the docs.

7b8bc8d

gspetro-NOAA reviewed Nov 14, 2024

View reviewed changes

doc/Makefile Outdated Show resolved Hide resolved

MichaelLueken approved these changes Nov 15, 2024

View reviewed changes

MichaelLueken added the run_we2e_coverage_tests Run the coverage set of SRW end-to-end tests label Nov 15, 2024

MichaelLueken linked an issue Nov 18, 2024 that may be closed by this pull request

Tech Doc Updates for Verification (PR #1137) #1145

Closed

Update doc/UsersGuide/BuildingRunningTesting/RunSRW.rst

72de59d

Co-authored-by: Gillian Petro <[email protected]>

MichaelLueken merged commit 39d1e5d into ufs-community:develop Nov 18, 2024
3 checks passed

MichaelLueken mentioned this pull request Nov 18, 2024

Updates to the authoritative develop branch since the SRW v2.2 release (10/31/2023) #981

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[develop] Make `get_obs` tasks day-dependent in workflow; other improvements and bug fixes #1137

[develop] Make `get_obs` tasks day-dependent in workflow; other improvements and bug fixes #1137

gsketefian commented Oct 8, 2024 •

edited

Loading

gspetro-NOAA commented Nov 13, 2024

MichaelLueken commented Nov 13, 2024

gsketefian commented Nov 14, 2024

gsketefian commented Nov 14, 2024

MichaelLueken commented Nov 14, 2024

gspetro-NOAA commented Nov 14, 2024 •

edited

Loading

maddenp-noaa commented Nov 14, 2024

MichaelLueken commented Nov 14, 2024

gsketefian commented Nov 15, 2024

MichaelLueken left a comment

MichaelLueken commented Nov 18, 2024

gsketefian commented Nov 20, 2024

[develop] Make get_obs tasks day-dependent in workflow; other improvements and bug fixes #1137

[develop] Make get_obs tasks day-dependent in workflow; other improvements and bug fixes #1137

Conversation

gsketefian commented Oct 8, 2024 • edited Loading

DESCRIPTION OF CHANGES:

Changes related to get_obs tasks:

Changes related to vx pre-processing tasks (PcpCombine_obs and Pb2nc_obs):

Small, self-contained bug fixes and improvements:

New WE2E tests added:

Type of change

TESTS CONDUCTED:

DOCUMENTATION:

CHECKLIST

LABELS (optional):

CONTRIBUTORS (optional):

gspetro-NOAA commented Nov 13, 2024

MichaelLueken commented Nov 13, 2024

gsketefian commented Nov 14, 2024

gsketefian commented Nov 14, 2024

MichaelLueken commented Nov 14, 2024

gspetro-NOAA commented Nov 14, 2024 • edited Loading

maddenp-noaa commented Nov 14, 2024

MichaelLueken commented Nov 14, 2024

gsketefian commented Nov 15, 2024

MichaelLueken left a comment

Choose a reason for hiding this comment

MichaelLueken commented Nov 18, 2024

gsketefian commented Nov 20, 2024

[develop] Make `get_obs` tasks day-dependent in workflow; other improvements and bug fixes #1137

[develop] Make `get_obs` tasks day-dependent in workflow; other improvements and bug fixes #1137

gsketefian commented Oct 8, 2024 •

edited

Loading

Changes related to `get_obs` tasks:

Changes related to vx pre-processing tasks (`PcpCombine_obs` and `Pb2nc_obs`):

gspetro-NOAA commented Nov 14, 2024 •

edited

Loading